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W.  H.  Fleming 

Progress  was  made  by  Fleming  and  associates  on  topics  in  stochastic  con¬ 
trol,  theory  of  viscosity  solutions  of  nonlinear  partial  differential  equations, 
asymptotic  problems  for  Markov  diffusion  processes  and  nonlinear  filtering. 

1.  Stochastic  control.  Fleming  and  Fitzpatrick  studied  finite  difference 
approximations  to  the  dynamic  programming  equation  for  some  particular 
classes  of  optimal  stochastic  control  problems.  These  arise  in  models  with 
two  types  of  control  actions.  One  represents  a  consumption  rate,  the  other  a 
rate  of  investment  or  exploration  for  an  undiscovered  natural  resource  (e.g. 
oil).  These  models  have  special  linear-concave  properties,  which  were  used  to 
prove  convergence  of  optimal  feedback  control  policies  as  the  mesh  size  tends 
to  zero.  In  a  different  direction  is  the  Ph.D.  research  of  H.  Zhu  on  singular 
stochastic  control  problems.  One  part  of  Zhu’s  thesis  extends  earlier  work 
of  Fleming  and  Vermes;  using  convex  duality  techniques.  Other  parts  of  the 
thesis  exploit  viscosity  solution  methods  to  study  a  broad  class  of  singular 
stochastic  control  problems. 

2.  Viscosity  solutions.  In  earlier  work,  Fleming  and  Souganidis  used 
viscosity  solution  methods  to  study  asymptotic  problems  (such  as  the  exit 
problem)  for  nearly  deterministic  controlled  Markov  diffusions.  In  a  1989 
paper,  they  considered  two-controller,  zero-sum  stochastic  differential  games, 
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and  characterized  the  value  function  in  terms  of  viscosity  solutions.  Recently, 
Fleming  and  McEneaney  used  viscosity  solution  methods  and  differential 
game  ideas  to  study  risk  sensitive  control  of  nonlinear  systems. 

3.  Asymptotic  problems  for  Markov  processes.  Fleming  and  James  ob¬ 
tained  much  sharper  versions  of  the  Freidlin-Wentzell  large  deviations  formu¬ 
las  for  exit  probabilities  for  nearly  deterministic  Markov  diffusion  processes. 
These  results  were  obtained  by  a  mixture  of  ideas  from  probability,  partial 
differential  equations  and  calculus  of  variations. 

4.  Nonlinear  filtering  with  small  observation  noise.  Fleming  and  Par- 
doux  developed  a  technique  which  provides  low-dimensional  approximations 
to  optimal  filters  (such  as  extended  Kalman  filters)  in  cases  where  a  piece- 
wise  one-to-one  function  of  a  system  state  plus  low  intensity  white  noise 
is  observed.  This  direction  of  work  was  continued  by  Fleming,  Zhang  and 
others.  They  dealt  with  sequential  hypothesis  tests,  which  are  superior  to 
the  fixed  time  statistical  tests  proposed  by  Fleming  and  Pardoux,  as  well  as 
numerical  aspects  of  the  problem. 
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1.  Convex  duality  approach  to  the  optimal  control  of  diffusions  (with  D. 
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2.  Value  functions  for  two-player,  zero-sum  stochastic  differential  games 
(with  P.  E.  Souganidis),  Indiana  University  Math  J.  3S  (1989)  293-312. 

3.  Asymptotic  expansions  for  Markov  processes  with  levy  generators  (with 
H.  M.  Soner),  Applied  Math,  and  Optimiz.  19  (19S9)  203-223. 

4.  Generalized  solutions  and  convex  duality  in  optimal  control  in  partial 
differential  equations  and  the  calculus  of  variations  (ed  F.  Colombini 
et  al),  Birkhauser,  1989,  461-472. 
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5.  Piecewise  monotone  filtering  with  small  observation  noise ,  (with  E. 
Pardoux)  SIAM  J.  on  Control  and  Optimiz.  27  (1989)  1156-1181. 

6.  Piecewise  monotone  filtering  in  discrete  time  with  small  observation 
noise,  (with  D.  Ji,  P.  Salame  and  Q.  Zhang)  IEEE  Trans.  Auto.  Con¬ 
trol  36  (1991)  1181-1185. 

7.  Numerical  method  for  an  optimal  investment/ consumption  problem, 
(with  B.  Fitzpatrick)  Math.  Oper.  Res.  16  (1991)  823-841. 

8.  Asymptotic  series  and  exit  time  probabilities,  (with  M.  James)  to  ap¬ 
pear  in  Annals  of  Probability. 

9.  Nonlinear  filtering  with  small  observation  noise:  piecewise  monotone 
observations,  (with  Q.  Zhang)  in  Stochastic  Analysis,  eds.  E.  Merzbach, 
A.  Schwartz  and  E.  Mayer- Wolf.  Academic  Press  (1991)  153-168. 

10.  Piecewise  monotone  filtering  with  small  observation  noise  (with  Q. 
Zhang)  Proc.  Joint  US  -  France  Workshop  on  Stochastic  Analysis  (eds. 
I.  Karatzas  and  D.  Ocone)  April  1991. 

11.  Risk  sensitive  optimal  control  and  differential  games  (with  w.  McE- 
neaney),  LCDS  Report  92-1. 

12.  Controlled  Markov  Processes  and  Viscosity  Solutions  (with  H.  M.  Soner) 
Springer  -  Verlag,  1992. 
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H.  J.  Kushner 


A  great  deal  of  work  was  done  in  the  area  of  numerical  methods  for 
stochastic  control  problems.  Among  the  most  challenging  problems  in  the 
stochastic  control  area  are  the  so-called  singular  control  problems:  They  arise 
in  numerous  areas;  for  example,  routing  questions  in  computer  and  commu¬ 
nications  systems.  We  developed  a  powerful  class  of  numerical  methods  for 
such  problems.  The  basic  models  used  are  diffusions  or  reflected  diffusions, 
but  the  method  is  a  general  applicability.  The  central  idea  is  that  of  the 
Markov  chain  approximation  method,  where  an  approximation  to  the  con¬ 
trol  problem  is  found  for  which  an  optimal  solution  is  computable,  and  which 
is  an  arbitrarily  good  approximation  to  the  original  problem  and  its  optimal 
value  function.  The  methods  are  convenient  to  program  and  use  (and  they 
have  been  used  with  success),  and  they  cover  a  wide  variety  of  problems. 
In  fact,  for  the  singular  problem,  they  seem  to  be  the  only  ones  currently 
available.  Owing  to  problems  in  proving  tightness  of  certain  processes  which 
occur  in  the  convergence  proofs,  the  methods  of  proof  used  for  the  non¬ 
singular  problems  need  modifications.  Examples  of  useful  approximations, 
the  algorithms  and  the  convergence  proofs  are  given.  In  order  to  illustrate 
the  power  of  the  methods,  two  classes  of  problems  are  dealt  with;  the  first 
is  a  class  of  discounted  problems,  and  the  second  is  an  average  cost  pe4  unit 
time  problem  subject  to  some  constraints,  and  which  arises  in  the  study  of 
multi-customer  class  queueing  networks  under  conditions  of  Ik  avy  traffic. 
The  method  is  applicable  to  the  more  standard  singular  control  and  ergodic 
problems  and  with  greater  ease.  The  algorithms  have  been  coded  and  tested 
on  problems  of  practical  interest.  They  work  well. 

Other  work  included  advances  in  the  use  of  Mome  Carlo  type  methods 
for  nonlinear  systems  modelled  by  diffusion  type  processes  and  where  the 
dimension  might  be  high.  Here  one  parmetrizes  the  control  a  priori,  and  seeks 
to  optimize  the  parameters,  for  the  appropriate  given  cost  of  performance 
function.  Often  the  performance  function  is  not  “nice;”  for  example,  it  might 
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be  the  probability  that  an  object  leaves  a  region  in  a  given  time  interval. 
This  function  is  not  nice  since  it  is  the  expectation  of  an  indicator  function, 
which  cannot  be  differentiated.  A  major  problem  in  Monte-Carlo  work  is 
the  estimation  of  the  derivatives  (with  respect  to  the  parameters)  of  the  cost 
function  from  simulated  samples.  Methods  for  doing  this  were  developed, 
based  on  using  sensitivity  functions  (measure  transformations).  The  method 
is  asymptotically  consistent,  and  yields  good  numerical  results.  The  ergodic 
cost  problem  was  also  dealt  with,  a  particularly  hard  problem  in  Monte-Carlo. 
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